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Abstract 


This paper reports on a study that explored the impacts of the Secondary 
Education Examination (SEE) English test on students’ motivation to learn 
English. Unlike previous studies on test impact, this study has extensively 
explored both the pre-test and post-test impacts of the test on students’ 
motivation to learn English. The data generated through a longitudinal 
survey among 247 SEE candidates and semi-structured interviews with six 
students and their parents in both the pre-test and post-test contexts indicates 
that the majority of students (79%) were motivated to learn English in the 
pre-test context but more than half of the students (54%) were discouraged 
from learning English in the post-test context. There were several reasons 
behind their demotivation to learn English in the post-test context including 
unfair judgement in the test and poor conduction of its speaking test. 
Pedagogical implications of the study have been discussed. 
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INTRODUCTION 


The research reported in this paper explored the impacts of the Secondary Education 
Examination (SEE) English, a high-stakes test in Nepal, on students’ motivation to learn 
English. It should be noted that the SEE was previously known as the School Leaving 
Certificate (SLC). Therefore, the terms SLC and SEE are used to refer to the same examination 
in this paper. The SEE serves several functions such as a gateway to higher education, a 
measure of students’ language skills, and a basic license for most official employment in Nepal 
(Dawadi & Shrestha, 2018). Additionally, the SEE results are also a measure of “what strengths 
and weaknesses exist in the education system at a given point in time, and how the education 
system is performing over the years” (Mathema & Bista, 2006, p.4). It is also worth pointing 
out the SEE English test is divided into two: speaking test (that includes listening and speaking 
skills) and writing test (that includes reading and writing skills). The speaking test is conducted 
by the schools themselves whereas the writing test is externally controlled by the National 
Exam Board, a constituent organization of the Ministry of Education in Nepal. 


However, almost no research has explored the impacts of the examination on students’ 
motivation to learn English, excluding Dawadi (2018). Dawadi’s research also does not record 
the voices of students as the data in her study was collected only from teachers. It is worth 
noting that if the impacts of a high-stakes test is not known, this might bring undesirable 
consequences to the society (Messick, 1998). It is potentially like “a police force without a 
court system, unfair and dangerous” (McNamara, 2007, p.280). Therefore, the rationale for this 
study begins with the need to explore the impacts of the SEE English test on students. However, 
it should be noted that this is a part of a research study that has explored a number of other 
issues related to the impacts of the SEE English test on students and their parents. 


Literature on language assessment indicates that the studies using students’ diaries and a 
longitudinal survey (a mixed methods approach) to explore the impact of a test on students are 
rare. This study, instead of capturing a snapshot of test preparation, followed the SEE 
candidates for several months, covering the whole process from initial preparation to receiving 
the test results to studying in a higher Grade. Characteristic of this study was to look at the test 
impacts and the test process from the students’ perspective and to give them a voice, rather 
than writing about them. Thus, this study seeks to make a methodological contribution to the 
existing research. A second potential contribution is to add more evidence of the test impacts 
on students. In terms of application, it can contribute to the improvement of the test and better 
inform teachers and policy makers about the nature of high-stakes test impacts on students and 
their parents. 


Test impact on student motivation 


Motivation has been defined as a person’s desire or drive to perform a particular task. It is 
usually considered to be one of the most important determinants of students’ success in high- 
stakes tests. However, it is a hugely complex phenomenon. Gardener (1985) defines 
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motivation as the extent to which an individual learner exerts effort to learn a language because 
of his or her desire to do so and the satisfaction derived from the task. Dérnyei (2005), a 
renowned figure in L2 motivation studies, considers motivation as one of the major factors that 
not only stimulates learners to initiate L2 learning but also equips them with the subsequent 
driving force to maintain the demanding and laborious L2 learning process. 


However, Ryan and Deci (2000) argue that as motivation is hardly a unitary concept, people 
vary not only in terms of the degree of motivation but also in the orientation (type) of 
motivation. Eccles and Wigfield (2002) present several motivation theories based on reasons 
for engagement in tasks (e.g., intrinsic motivation theory, flow theory, self-determination 
theory, and goals theory); theories that integrate expectancy and value constructs (e.g., self- 
worth theory, expectancy-value theory, and attribution theory); and theories that focus on 
integrating motivation and cognition (e.g., theories of motivation and volition, and theories of 
self-regulation and motivation). A full discussion of those theories is beyond the capacity of 
this paper. 


As the main concern of the current study is on whether or not the SEE English test motivates 
students to learn the English language, it focuses more on the self-determination theory (SDT) 
of motivation which mainly focuses on human beings’ natural tendencies to behave in an 
effective way (Cheng et al., 2014). SDT was first developed by Richard M. Ryan and Edward 
L. Deci who later developed the notions of extrinsic and intrinsic motivation (Ryan & Deci, 
2000). SDT, depending on the degree of self-determination, puts human motivation on a 
continuum ranging from amotivation (the state of lacking willingness to act) to passive 
compliance (i.e. controlled forms of extrinsic motivation) to active personal commitment (i.e. 
intrinsic motivation) (Ryan & Deci, 2000). In a high-stakes test context, “motivation to succeed 
is not static, but occurs along a continuum. In particular, motivation varies according to the 
complex interaction of test-takers and test contexts based on both the intended and unintended 
test use” (Cheng, et al., 2014, p.306). However, Ryan and Brown (2005) claim that high-stakes 
testing policies are mostly developed based on the notion that punishments, rewards and self- 
esteem-based pressures function as effective motivators for learning. It is usually believed that 
tests provide incentives to test-takers to improve their performances. Nonetheless, tests might 
be more motivating for those test-takers who expect success. 


Gardner (2001) uses the terms ‘integrative’ and ‘instrumental’ to classify motivation. The term 
‘integrative’ is synonymous to intrinsic motivation and instrumental to extrinsic motivation. 
Intrinsic motivation refers to the context when a person is doing something as s/he finds it 
inherently interesting or enjoyable, whereas extrinsic motivation refers to the context when the 
person is performing an action as s/he thinks that learning will be beneficial for future career 
(Ryan & Deci, 2000). An intrinsically motivated person shows interest in learning about the 
people of the target language and their culture and engages in an activity and also enjoys 
learning the language (Gardner, 2001). However, an instrumentally or extrinsically motivated 
learner has more pragmatic considerations or instrumental purpose for L2 learning, such as 
getting a reward, earning money or obtaining a job. Learning for a test is generally regarded as 
instrumental, which is normally an extrinsic motive for students (Zhan & Andrews, 2014). 
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Kwon, Lee & Shin (2017) and Choi (2008), based on their research in the Korean context, 
argue that tests lead to low intrinsic motivation in students as most students’ drive and 
motivation for learning is closely linked to instrumental motivation such as getting admission 
in a good university. 


Testing researchers seem to hold different views regarding test motivation. For instance, Hsu 
(2009) argues that students in high-stakes test contexts are less likely to hold positive attitudes 
towards learning as the performances in those tests are likely to determine their career or lives. 
Additionally, Tsai and Tsou (2009) claim that high-stakes tests lead to a decrease in motivation 
to learn English as classes are test-oriented, enhancing only test-taking skills instead of 
developing communicative competence. However, Abu-Alhija (2007) opines that large-scale 
tests in some contexts may motivate students to work harder and more effectively. Some 
previous washback studies (e.g. Dawadi, 2018; Shohamy, Donitsa-Schmidt, & Ferman, 1996) 
also claim that high-stakes English tests are a strong instrumental motivation for students to 
learn English. However, very little is known about the impacts of the SEE English test on 
students’ motivation to learn English. Therefore, this study aims to explore how the test affects 
students’ motivation to learn both in the pre-test and post-test contest. 


Participants 


The participants of the study included secondary level students (N=247) studying at Grade 10 
in public schools in Nepal. They were 14 to 16 years old and they had learned English for 10 
years. All of them were Nepali native speakers and had learnt English as a foreign language. 
They were preparing for the SEE at the beginning phase of this research project but they were 
studying at Grade 11 during the final phase of the data collection. 


Research ethics 


The institutional code of practice for research and British Educational Research Association 
(BERA) ethical guidelines (2018) were followed to maintain the ethics in this study. Both 
students and parents were given a full account of the project before they took part in the study. 
They were informed that their participation in this study was voluntary and that all data would 
be treated as confidential and anonymised promptly. Parents’ consent was obtained before 
students took part in this study. 


Data Collection 


This study featured a mixed method design comprising a longitudinal survey, oral diaries and 
interviews. The first questionnaire survey was carried out about six weeks before the students 
sat for the SEE and the second survey was conducted two months after the SEE results 
publication. It should be noted that the number of respondents in the pre-test survey was 247, 
whereas it was 226 in the post-test survey. Additionally, six case study students were asked to 
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record weekly oral diaries for three months: first during the usual classes (i.e. in the fifth month 
of their academic year), second during the test preparation (in the ninth month of the academic 
year) and the final one around the test results (1.e., they started to record their diaries one week 
before the test results and continued for four weeks). Furthermore, the case study students were 
interviewed twice: around six weeks before the SEE and two months after the SEE result 
publication. Considering the English language proficiency of the participants, only the Nepali 
language was used for the data collection. 


Data Analysis 


The quantitative data was analysed using SPSS version 20 and frequency distributions were 
calculated for each item included in the questionnaire. In some cases, central tendency statistics 
(i.e. mean percentage) has been reported to indicate an overall picture of the students’ views 
on the nature of the test impacts. In order to analyse qualitative data, NVivo 10 was used and 
a thematic analysis approach was employed. Thus, qualitative data analysis was an iterative 
process which included several readings and rereading of the transcribed texts. 


FINDINGS 


Before presenting the findings, it is worth pointing out that each case study student and parent 
is represented with the alphabets ‘S’ and ‘P’ respectively, followed by a number | to 6 to ensure 
confidentiality in this research. The same number is used to represent a student and his/her 
parent. For instance, Pl means only the parent of S1 and P2 means only the parent of S2 and 
so on. 


The study explored both the pre-test and post-test motivation to learn English. Therefore, 
students were asked to respond to the same question in both the surveys and similar questions 
were asked in both the pre-test and post-test interviews. Students’ responses to the 
questionnaires are summarised in Figure | which indicates that the majority of students (79%) 
were motivated to learn English in the pre-test context. However, there was a sharp decrease 
in the number of students who were motivated to learn English in the post-test context. Only 
30% students were motivated to learn English and slightly more than half of the students (52%) 
were even discouraged from learning English in the post-test context. Nevertheless, there was 
no effect of the test on some students, both in the pre-test and post-test contexts. 
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Test impact on student motivation to learn English (N=247 pre-test, 226 
post-test) 


motivates me to learn discourages me from __ no effects on my 
English learning English motivation to learn 
English 
pre-test Mpost-test 


Figure 1: Test impact on student motivation to learn English 


With regard to the pre-test motivation to learn English, the quantitative results were well 
supported by the qualitative findings. Test motivation for learning English was one of the most 
frequently occurring sub-themes in the qualitative data analysis. Most of the pre-test diaries 
indicated that the students were motivated to learn English and do well on the test: “Last week, 
I was motivated to learn English as I knew that the SEE is very important for my future” (S3); 
“Tam really interested in learning English these days as the SEE is coming soon. I know that 
the SEE is very important for me. I must try my best to do well on the test” (S1). Additionally, 
all the parents, during their pre-test interviews, reported that the test motivated their children 
to learn English: “7 think, she is more motivated to learn English this year because of the SEE” 
(P1). Furthermore, all the case study students reported that their parents would motivate them 
to learn English and work hard for the test: “My mother always encourages me to learn 
English” (S5). 


Interestingly, it was also found that one of the case study students (S5), who did not like the 
English subject before she started to study at Grade 10, got interest in it and also had a sense 
of achievement when she prepared for the SEE English test. 


When I was at Grade 9, I was never interested in learning English as I found the subject 
very difficult. This year, I am a bit more interested in this subject. I started to take 
tuition classes right after I began to study at Grade 10. I spent more time for this 
subject. Now, I have somehow learnt English and also performed better in our class 
tests. I have realised that English is not a very difficult subject, if we work hard (SS). 
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The findings indicated that there were two main reasons for their motivation to do well on the 
test: they had a plan to continue their study after the SEE and wished to study in a good higher 
secondary school or college of their choice: 


I am very much motivated to learn English because I need to do well on the test to be 
eligible to continue my study. Also, I need to get minimum C+ grade on the test to be 
eligible to study science at Grade 11(S3). 


I want to work hard for the test as I want to study in a good higher secondary school in 
Kathmandu (S3). 


I know that good colleges enrol only the students who have got good Grade on the test 
(S1). 


The findings further indicated that the students had a belief that doing well on the test would 
support them to find a job in future: “Jf I do well on the test, it helps me to find a good job in 
future” (S5). Thus, the test use seems to be one of the motivating factors for them. 


Interestingly, two of the students (S4, S6) were also interested in learning about English people 
and their culture: “J am interested in learning about English people and their culture.” (S6). 


The diaries also indicated that doing well on their class tests would sometimes be a source of 
motivation for them: “Last week, I was really interested in learning English because I got good 
scores in our Class test. My teacher praised me in front of my classmates that made me feel 
proud” (S4). Conversely, not being able to perform well on those tests would discourage them 
from learning English: “7 performed poorly on the test last week. So, I was just worried and 
could not study much” (S2). 


With regard to the post-test motivation, it might be logical to divide it into two sections on the 
basis of the timing (i.e. before and after the test results publication). It should be noted that the 
quantitative data presented in Figure | represents only post-result motivation. 


The quantitative finding that just above half of the surveyed students (52%) were discouraged 
from learning English after the test results publication. The findings were supported by the 
qualitative findings. Three of the case study students reported that the test results discouraged 
them from learning English: “J am really discouraged from leaning English these days. I really 
feel bad as I could not get my expected Grades” (S1); “I am very much discouraged from 
learning English now. I am particularly not happy with my grade on the speaking test. I do not 
think that the test was fair-S3). 


Nevertheless, rest of the students reported that they were more motivated to lean English 
because of the test results: “With a great surprise, I got better grades than I had expected. I 
feel proud and more motivated to learn English now” (S4); “The test results have really 
encouraged me to work hard” (S5). 


The findings related to S2 were more interesting. She was motivated to learn English during 
the first phase of diary recording, then she was discouraged from learning English when the 
test grew closer mainly because of the perceived difficulty of the test: 
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English is the most difficult subject for me. Although I try my best, I cannot get good 
scores on our Class tests. I have heard that the SEE will be more difficult than our class 
tests. Therefore, I sometimes feel really discouraged from learning English (S2). 


She was again motivated to learn English after the test results as she obtained higher Grade 
than her expectation: “J got higher Grade on my speaking test that made me happy and also 
encouraged me to lean English” (S2). Indeed, the excerpt indicates that successful performance 
on the test became a motivating factor to learn English after the results. 


The diaries recorded just one week before the test result indicated that three students were 
learning English during the time. Among them two students were living in a city and preparing 
for their college entrance exam as both of them had a plan to apply for a reputed college in a 
city. They were also taking a bridge course and trying to develop their language skills: “J spent 
the whole week in reading and writing as I had to go through a very tough competition to get 
admission in a good college” (S1); “I also practised speaking English with my friends in my 
bridge course” (S3). 


However, their interviews suggested that their motivation to develop language skills was 
triggered by the format of the college entrance exam that would include a large sample of 
writing such as essay writing, report writing and letter writing, as indicated by the following 
excerpt: 


I basically tried hard to develop my writing skills during the time as I knew that there 
would be different kinds of free writing questions in our college entrance test (S1). 


An additional factor of their motivation was the medium of instruction at Grade 11: 


All the subjects are taught in English at Grade 11. Therefore, I tried my best to develop 
my speaking skills as I had not practised it before (S3). 


Nevertheless, three other students did almost nothing in English during the time, except 
occasionally listening to English songs: “J have done nothing in English after I took the test. I 
will restart learning it after I begin to study at Grade I1” (SS). 


To sum up, having brought both the quantitative and qualitative data together, this section has 
produced a comprehensive picture of the impacts of the test on students’ motivation. The 
quantitative results were well supported by qualitative findings, and vice versa. Students’ 
diaries and interviews provided further explanations for the reasons of their motivation. Indeed, 
students’ diaries have beautifully captured how students’ motivation level kept changing during 
different stages of the academic year. In other words, had the students not been asked to record 
their diaries, it would not have been possible to capture students’ motivation level through the 
pre-test context to the post-test context to the post-results context. 


DISCUSSION 


With regard to the pre-test motivation to learn English, the findings of the study indicate that 
the students perceived the test impact as more positive than negative. These findings are 
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consistent with the findings of previous washback studies (e.g. Dawadi, 2018; Gu, 2005; 
Huang, 2010; Li, et al. 2012; Takagi, 2010) that taking or preparing for a high-stakes EFL test 
usually motivates students to learn English. 


There could be possibly two reasons behind students’ motivation to learn English in the pre- 
test context. First, the test results are used as a gateway to higher education and also a criterion 
for subject selection in higher secondary education. This test policy seems to be effective to 
motivate students. Second, the SEE English test results may affect their chances to study in the 
higher secondary schools/colleges of their choices and to find a job. 


Indeed, previous research also indicates that students who consider a test to be important for 
them tend to study hard for it (Allen, 2016; Cheng, 1997; Chu et al., 2014). In other words, if 
students know that their test performance will have significant effects on their career goals, it 
is more likely that they expend greater efforts to perform well on the test. 


The findings emerged through the qualitative data collected from S2 (that she was discouraged 
from learning English when the test grew closer mainly because of the perceived difficulty of 
the test but she was again motivated to learn English after she got good results in the SEE) 
provide further evidence to the claim that assessment can motivate students when results are 
good (Xiao & Carless, 2013). Additionally, the findings might indicate that students’ 
motivation level does not remain constant and they lose interest in learning if they perceive a 
test to be very difficult for them. 


Contrary to Shohamy, et al.’s (1996) claim that high-stakes EFL test-takers’ motivation to 
learn English significantly decreases after the test administration, half of the case study students 
(i.e., three out of six) in the current study were still motivated to learn English. They were 
actively learning English in the post-test context (i.e. about three months when they were 
waiting for the test results) and trying their best to develop their language skills. However, 
college entrance exam, rather than the SEE English test, seemed to be the key motivating factor 
for them to learn English during the time. It can be assumed that if they were not supposed to 
take the exam, they might not have been motivated to learn English during the time. However, 
a few students were discouraged from learning English in the post-test context as they did not 
get their expected grade on the test. This means, they did not think that the test was fair. 


It is also worth noting that four of the case study students in this study were instrumentally 
motivated to learn English. Very similar to Greek students (Tsagiri, 2006), incentive values 
and instrumental benefits were a stronger motivation for learning English than their interest in 
learning about cultural artefacts, such as English movies, books, newspapers and magazines, 
or learning about English speakers and their culture. More specifically, the main reason for 
their motivation was that English would help them to make their future bright due to its 
importance, popularity and usefulness in the Nepalese context. However, the rest of the 
students were intrinsically motivated to learn English. 


As reported by Tsagiri (2006), the initial driving force of their motivation to learn English 
seemed to be influenced by the social milieu, especially their parents. All the case study 
students reported that their parents motivated them to learn English. The general perception 
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that there is a great role of family in motivating Asian students (Life, 2011) is well supported 
by this study. 


No matter what the source of motivation was for the SEE students, what was more important 
for this study was that the students were highly motivated to perform well on the test. It seemed 
that the SEE was perceived as a goal which gave them a direction for promoting their learning. 
Studying for the SEE also seemed to help students develop good learning habits, such as getting 
up early and doing homework every day. 


To sum up, students’ motivation level does not remain constant; generally it was stronger in 
the pre-test context than in the post-test context. The main reason behind this could be that the 
reward for the students who succeed in the SEE is potentially very positive, with some very 
attractive career and life-altering possibilities. 


Limitations of the study 


The study has revealed interesting findings regarding the impacts of the SEE English test on 
student motivation to learn English. However, the results of this study should be treated with 
caution and future investigation should try to overcome the limitations of this study. To be 
more specific, the study has three major limitations. The first limitation of the study concerns 
its sample size. It was limited to 247 students and six parents in Nepal. Therefore, 
generalisation of the findings is limited by this constraint. 


The second limitation of this study is from a methodological point of view. This study was 
limited to the data collected from students and their parents, but it would have been benefited 
more from the data collected from schools or from teachers. Finally, all the participants in this 
study were selected only from public schools but around 20% students in Nepal study in private 
schools. Therefore, this study cannot represent the voices of private school students. This study, 
however, is one of the few test impact studies that has explored the impacts of a high-stakes 
test on students both in the pre-test and post-test contexts. 


Implications of the Study 


There are two major implications of the study. Firstly, the findings of the study indicated that 
students were motivated to learn English in the pre-test context. However, they were 
discouraged form learning English in the post-test context. There were several indications that 
one of the main reasons of their discouragement was the test fairness. This means, some of the 
students did not believe that the test (particularly speaking test) was fair and could accurately 
measure their language skills. Thus, a due attention need be paid on the test design. Secondly, 
the finding of the study that students’ motivation to learn the English did not remain constant 
might encourage test impact researchers to carry out a longitudinal study rather than limiting 
their studies to test preparation phase. 
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Recommendations for Future Research 


This study collected data only from students and their parents. However, it would have been 
definitely helpful to know about teachers’ views. Had the data from teachers been obtained, it 
might have helped to clarify several ambiguities emerged in this data that would produce a 
more comprehensive picture. Thus, this argument highlights that methodological triangulation 
in test impact research like this is crucial. It is hoped that future research takes care of such 
issues and includes methodological triangulation to create more comprehensive picture of the 
test impacts. Furthermore, this study was limited to public school students and their parents 
from a remote part of Nepal. Therefore, it is recommended that future research includes 
students from both private and public schools and also from city areas. 


While reviewing literature for this study, it was found that almost all the previous test impact 
studies (excluding Loumbourd, 2014) observed only the pre-test impacts of high-stakes tests 
on students. However, this study has revealed that a test can affect students’ motivation to learn 
both in pre-test and post-test contexts and such impact does not remain constant. Therefore, it 
is recommended that future research is not limited to the test preparation and explores both the 
pre-test and post-test impacts of a high-stakes test on students. 


CONCLUSION 


This study explored the impacts of the SEE English test on students’ motivation to learn 
English. The data generated through multiple sources indicate that the test had considerable 
impact on students’ motivation to learn English both in the pre-test and post-test context; pre- 
test impact was more positive than post-test impact. Although the test motivated a large number 
of students in the pre-test context, more than half of the students were discouraged from 
learning English in the post-test context. One of the main reasons behind students’ 
dissatisfaction was that they did not find the test as fair as they expected. Thus, this study has 
raised several questions about the reliability and validity of the test. Having collected both 
students’ and their parents’ views about the SEE, this study has illuminated the areas which are 
not fully explored. 
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